{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous training pipeline with KFP and Cloud AI Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning Objectives:**\n",
    "1. Learn how to use KF pre-build components (BiqQuery, CAIP training and predictions)\n",
    "1. Learn how to use KF lightweight python components\n",
    "1. Learn how to build a KF pipeline with these components\n",
    "1. Learn how to compile, upload, and run a KF pipeline with the command line\n",
    "\n",
    "\n",
    "In this lab, you will build, deploy, and run a KFP pipeline that orchestrates **BigQuery** and **Cloud AI Platform** services to train, tune, and deploy a **scikit-learn** model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PATH=/home/jupyter/.local/bin:/usr/local/cuda/bin:/opt/conda/bin:/opt/conda/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games\n"
     ]
    }
   ],
   "source": [
    "# Set `PATH` to include the directory containing TFX CLI and skaffold.\n",
    "PATH=%env PATH\n",
    "%env PATH=/home/jupyter/.local/bin:{PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the pipeline design\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The workflow implemented by the pipeline is defined using a Python based Domain Specific Language (DSL). The pipeline's DSL is in the `covertype_training_pipeline.py` file that we will generate below.\n",
    "\n",
    "The pipeline's DSL has been designed to avoid hardcoding any environment specific settings like file paths or connection strings. These settings are provided to the pipeline code through a set of environment variables.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BASE_IMAGE = os.getenv('BASE_IMAGE')\n",
      "TRAINER_IMAGE = os.getenv('TRAINER_IMAGE')\n",
      "RUNTIME_VERSION = os.getenv('RUNTIME_VERSION')\n",
      "PYTHON_VERSION = os.getenv('PYTHON_VERSION')\n",
      "COMPONENT_URL_SEARCH_PREFIX = os.getenv('COMPONENT_URL_SEARCH_PREFIX')\n",
      "USE_KFP_SA = os.getenv('USE_KFP_SA')\n"
     ]
    }
   ],
   "source": [
    "!grep 'BASE_IMAGE =' -A 5 pipeline/covertype_training_pipeline.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pipeline uses a mix of custom and pre-build components.\n",
    "\n",
    "- Pre-build components. The pipeline uses the following pre-build components that are included with the KFP distribution:\n",
    "    - [BigQuery query component](https://github.com/kubeflow/pipelines/tree/0.2.5/components/gcp/bigquery/query)\n",
    "    - [AI Platform Training component](https://github.com/kubeflow/pipelines/tree/0.2.5/components/gcp/ml_engine/train)\n",
    "    - [AI Platform Deploy component](https://github.com/kubeflow/pipelines/tree/0.2.5/components/gcp/ml_engine/deploy)\n",
    "- Custom components. The pipeline uses two custom helper components that encapsulate functionality not available in any of the pre-build components. The components are implemented using the KFP SDK's [Lightweight Python Components](https://www.kubeflow.org/docs/pipelines/sdk/lightweight-python-components/) mechanism. The code for the components is in the `helper_components.py` file:\n",
    "    - **Retrieve Best Run**. This component retrieves a tuning metric and hyperparameter values for the best run of a AI Platform Training hyperparameter tuning job.\n",
    "    - **Evaluate Model**. This component evaluates a *sklearn* trained model using a provided metric and a testing dataset.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./pipeline/covertype_training_pipeline.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./pipeline/covertype_training_pipeline.py\n",
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\"\"\"KFP pipeline orchestrating BigQuery and Cloud AI Platform services.\"\"\"\n",
    "\n",
    "import os\n",
    "\n",
    "from helper_components import evaluate_model\n",
    "from helper_components import retrieve_best_run\n",
    "from jinja2 import Template\n",
    "import kfp\n",
    "from kfp.components import func_to_container_op\n",
    "from kfp.dsl.types import Dict\n",
    "from kfp.dsl.types import GCPProjectID\n",
    "from kfp.dsl.types import GCPRegion\n",
    "from kfp.dsl.types import GCSPath\n",
    "from kfp.dsl.types import String\n",
    "from kfp.gcp import use_gcp_secret\n",
    "\n",
    "# Defaults and environment settings\n",
    "BASE_IMAGE = os.getenv('BASE_IMAGE')\n",
    "TRAINER_IMAGE = os.getenv('TRAINER_IMAGE')\n",
    "RUNTIME_VERSION = os.getenv('RUNTIME_VERSION')\n",
    "PYTHON_VERSION = os.getenv('PYTHON_VERSION')\n",
    "COMPONENT_URL_SEARCH_PREFIX = os.getenv('COMPONENT_URL_SEARCH_PREFIX')\n",
    "USE_KFP_SA = os.getenv('USE_KFP_SA')\n",
    "\n",
    "TRAINING_FILE_PATH = 'datasets/training/data.csv'\n",
    "VALIDATION_FILE_PATH = 'datasets/validation/data.csv'\n",
    "TESTING_FILE_PATH = 'datasets/testing/data.csv'\n",
    "\n",
    "# Parameter defaults\n",
    "SPLITS_DATASET_ID = 'splits'\n",
    "HYPERTUNE_SETTINGS = \"\"\"\n",
    "{\n",
    "    \"hyperparameters\":  {\n",
    "        \"goal\": \"MAXIMIZE\",\n",
    "        \"maxTrials\": 6,\n",
    "        \"maxParallelTrials\": 3,\n",
    "        \"hyperparameterMetricTag\": \"accuracy\",\n",
    "        \"enableTrialEarlyStopping\": True,\n",
    "        \"params\": [\n",
    "            {\n",
    "                \"parameterName\": \"max_iter\",\n",
    "                \"type\": \"DISCRETE\",\n",
    "                \"discreteValues\": [500, 1000]\n",
    "            },\n",
    "            {\n",
    "                \"parameterName\": \"alpha\",\n",
    "                \"type\": \"DOUBLE\",\n",
    "                \"minValue\": 0.0001,\n",
    "                \"maxValue\": 0.001,\n",
    "                \"scaleType\": \"UNIT_LINEAR_SCALE\"\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# Helper functions\n",
    "def generate_sampling_query(source_table_name, num_lots, lots):\n",
    "    \"\"\"Prepares the data sampling query.\"\"\"\n",
    "\n",
    "    sampling_query_template = \"\"\"\n",
    "         SELECT *\n",
    "         FROM \n",
    "             `{{ source_table }}` AS cover\n",
    "         WHERE \n",
    "         MOD(ABS(FARM_FINGERPRINT(TO_JSON_STRING(cover))), {{ num_lots }}) IN ({{ lots }})\n",
    "         \"\"\"\n",
    "    query = Template(sampling_query_template).render(\n",
    "        source_table=source_table_name, num_lots=num_lots, lots=str(lots)[1:-1])\n",
    "\n",
    "    return query\n",
    "\n",
    "\n",
    "# Create component factories\n",
    "component_store = kfp.components.ComponentStore(\n",
    "    local_search_paths=None, url_search_prefixes=[COMPONENT_URL_SEARCH_PREFIX])\n",
    "\n",
    "bigquery_query_op = component_store.load_component('bigquery/query')\n",
    "mlengine_train_op = component_store.load_component('ml_engine/train')\n",
    "mlengine_deploy_op = component_store.load_component('ml_engine/deploy')\n",
    "retrieve_best_run_op = func_to_container_op(\n",
    "    retrieve_best_run, base_image=BASE_IMAGE)\n",
    "evaluate_model_op = func_to_container_op(evaluate_model, base_image=BASE_IMAGE)\n",
    "\n",
    "\n",
    "@kfp.dsl.pipeline(\n",
    "    name='Covertype Classifier Training',\n",
    "    description='The pipeline training and deploying the Covertype classifierpipeline_yaml'\n",
    ")\n",
    "def covertype_train(project_id,\n",
    "                    region,\n",
    "                    source_table_name,\n",
    "                    gcs_root,\n",
    "                    dataset_id,\n",
    "                    evaluation_metric_name,\n",
    "                    evaluation_metric_threshold,\n",
    "                    model_id,\n",
    "                    version_id,\n",
    "                    replace_existing_version,\n",
    "                    hypertune_settings=HYPERTUNE_SETTINGS,\n",
    "                    dataset_location='US'):\n",
    "    \"\"\"Orchestrates training and deployment of an sklearn model.\"\"\"\n",
    "\n",
    "    # Create the training split\n",
    "    query = generate_sampling_query(\n",
    "        source_table_name=source_table_name, num_lots=10, lots=[1, 2, 3, 4])\n",
    "\n",
    "    training_file_path = '{}/{}'.format(gcs_root, TRAINING_FILE_PATH)\n",
    "\n",
    "    create_training_split = bigquery_query_op(\n",
    "        query=query,\n",
    "        project_id=project_id,\n",
    "        dataset_id=dataset_id,\n",
    "        table_id='',\n",
    "        output_gcs_path=training_file_path,\n",
    "        dataset_location=dataset_location)\n",
    "\n",
    "    # Create the validation split\n",
    "    query = generate_sampling_query(\n",
    "        source_table_name=source_table_name, num_lots=10, lots=[8])\n",
    "\n",
    "    validation_file_path = '{}/{}'.format(gcs_root, VALIDATION_FILE_PATH)\n",
    "\n",
    "    create_validation_split = bigquery_query_op(\n",
    "        query=query,\n",
    "        project_id=project_id,\n",
    "        dataset_id=dataset_id,\n",
    "        table_id='',\n",
    "        output_gcs_path=validation_file_path,\n",
    "        dataset_location=dataset_location)\n",
    "\n",
    "    # Create the testing split\n",
    "    query = generate_sampling_query(\n",
    "        source_table_name=source_table_name, num_lots=10, lots=[9])\n",
    "\n",
    "    testing_file_path = '{}/{}'.format(gcs_root, TESTING_FILE_PATH)\n",
    "\n",
    "    create_testing_split = bigquery_query_op(\n",
    "        query=query,\n",
    "        project_id=project_id,\n",
    "        dataset_id=dataset_id,\n",
    "        table_id='',\n",
    "        output_gcs_path=testing_file_path,\n",
    "        dataset_location=dataset_location)\n",
    "\n",
    "    # Tune hyperparameters\n",
    "    tune_args = [\n",
    "        '--training_dataset_path',\n",
    "        create_training_split.outputs['output_gcs_path'],\n",
    "        '--validation_dataset_path',\n",
    "        create_validation_split.outputs['output_gcs_path'], '--hptune', 'True'\n",
    "    ]\n",
    "\n",
    "    job_dir = '{}/{}/{}'.format(gcs_root, 'jobdir/hypertune',\n",
    "                                kfp.dsl.RUN_ID_PLACEHOLDER)\n",
    "\n",
    "    hypertune = mlengine_train_op(\n",
    "        project_id=project_id,\n",
    "        region=region,\n",
    "        master_image_uri=TRAINER_IMAGE,\n",
    "        job_dir=job_dir,\n",
    "        args=tune_args,\n",
    "        training_input=hypertune_settings)\n",
    "\n",
    "    # Retrieve the best trial\n",
    "    get_best_trial = retrieve_best_run_op(\n",
    "            project_id, hypertune.outputs['job_id'])\n",
    "\n",
    "    # Train the model on a combined training and validation datasets\n",
    "    job_dir = '{}/{}/{}'.format(gcs_root, 'jobdir', kfp.dsl.RUN_ID_PLACEHOLDER)\n",
    "\n",
    "    train_args = [\n",
    "        '--training_dataset_path',\n",
    "        create_training_split.outputs['output_gcs_path'],\n",
    "        '--validation_dataset_path',\n",
    "        create_validation_split.outputs['output_gcs_path'], '--alpha',\n",
    "        get_best_trial.outputs['alpha'], '--max_iter',\n",
    "        get_best_trial.outputs['max_iter'], '--hptune', 'False'\n",
    "    ]\n",
    "\n",
    "    train_model = mlengine_train_op(\n",
    "        project_id=project_id,\n",
    "        region=region,\n",
    "        master_image_uri=TRAINER_IMAGE,\n",
    "        job_dir=job_dir,\n",
    "        args=train_args)\n",
    "\n",
    "    # Evaluate the model on the testing split\n",
    "    eval_model = evaluate_model_op(\n",
    "        dataset_path=str(create_testing_split.outputs['output_gcs_path']),\n",
    "        model_path=str(train_model.outputs['job_dir']),\n",
    "        metric_name=evaluation_metric_name)\n",
    "\n",
    "    # Deploy the model if the primary metric is better than threshold\n",
    "    with kfp.dsl.Condition(eval_model.outputs['metric_value'] > evaluation_metric_threshold):\n",
    "        deploy_model = mlengine_deploy_op(\n",
    "        model_uri=train_model.outputs['job_dir'],\n",
    "        project_id=project_id,\n",
    "        model_id=model_id,\n",
    "        version_id=version_id,\n",
    "        runtime_version=RUNTIME_VERSION,\n",
    "        python_version=PYTHON_VERSION,\n",
    "        replace_existing_version=replace_existing_version)\n",
    "\n",
    "    # Configure the pipeline to run using the service account defined\n",
    "    # in the user-gcp-sa k8s secret\n",
    "    if USE_KFP_SA == 'True':\n",
    "        kfp.dsl.get_pipeline_conf().add_op_transformer(\n",
    "              use_gcp_secret('user-gcp-sa'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The custom components execute in a container image defined in `base_image/Dockerfile`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FROM gcr.io/deeplearning-platform-release/base-cpu\n",
      "RUN pip install -U fire scikit-learn==0.20.4 pandas==0.24.2 kfp==0.2.5\n"
     ]
    }
   ],
   "source": [
    "!cat base_image/Dockerfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training step in the pipeline employes the AI Platform Training component to schedule a  AI Platform Training job in a custom training container. The custom training image is defined in `trainer_image/Dockerfile`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FROM gcr.io/deeplearning-platform-release/base-cpu\n",
      "RUN pip install -U fire cloudml-hypertune scikit-learn==0.20.4 pandas==0.24.2\n",
      "WORKDIR /app\n",
      "COPY train.py .\n",
      "\n",
      "ENTRYPOINT [\"python\", \"train.py\"]\n"
     ]
    }
   ],
   "source": [
    "!cat trainer_image/Dockerfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and deploying the pipeline\n",
    "\n",
    "Before deploying to AI Platform Pipelines, the pipeline DSL has to be compiled into a pipeline runtime format, also refered to as a pipeline package.  The runtime format is based on [Argo Workflow](https://github.com/argoproj/argo), which is expressed in YAML. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure environment settings\n",
    "\n",
    "Update  the below constants  with the settings reflecting your lab environment. \n",
    "\n",
    "- `REGION` - the compute region for AI Platform Training and Prediction\n",
    "- `ARTIFACT_STORE` - the GCS bucket created during installation of AI Platform Pipelines. The bucket name starts with the `hostedkfp-default-` prefix.\n",
    "- `ENDPOINT` - set the `ENDPOINT` constant to the endpoint to your AI Platform Pipelines instance. Then endpoint to the AI Platform Pipelines instance can be found on the [AI Platform Pipelines](https://console.cloud.google.com/ai-platform/pipelines/clusters) page in the Google Cloud Console.\n",
    "\n",
    "1. Open the *SETTINGS* for your instance\n",
    "2. Use the value of the `host` variable in the *Connect to this Kubeflow Pipelines instance from a Python client via Kubeflow Pipelines SKD* section of the *SETTINGS* window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "ENDPOINT = '2154b2f21283d4d2-dot-europe-west1.pipelines.googleusercontent.com'\n",
    "ARTIFACT_STORE_URI = 'gs://qwiklabs-gcp-01-3aff5ef1f764-kubeflowpipelines-default'\n",
    "\n",
    "PROJECT_ID = !(gcloud config get-value core/project)\n",
    "PROJECT_ID = PROJECT_ID[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the trainer image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_NAME='trainer_image'\n",
    "TAG='latest'\n",
    "TRAINER_IMAGE='gcr.io/{}/{}:{}'.format(PROJECT_ID, IMAGE_NAME, TAG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating temporary tarball archive of 2 file(s) totalling 3.4 KiB before compression.\n",
      "Uploading tarball of [trainer_image] to [gs://qwiklabs-gcp-01-3aff5ef1f764_cloudbuild/source/1606483297.219497-fc73065cc7c24588b5b7cbe4afb8f50c.tgz]\n",
      "Created [https://cloudbuild.googleapis.com/v1/projects/qwiklabs-gcp-01-3aff5ef1f764/builds/5deceace-805d-406d-8e9e-bdfec88a4749].\n",
      "Logs are available at [https://console.cloud.google.com/cloud-build/builds/5deceace-805d-406d-8e9e-bdfec88a4749?project=1016448934670].\n",
      "----------------------------- REMOTE BUILD OUTPUT ------------------------------\n",
      "starting build \"5deceace-805d-406d-8e9e-bdfec88a4749\"\n",
      "\n",
      "FETCHSOURCE\n",
      "Fetching storage object: gs://qwiklabs-gcp-01-3aff5ef1f764_cloudbuild/source/1606483297.219497-fc73065cc7c24588b5b7cbe4afb8f50c.tgz#1606483297852569\n",
      "Copying gs://qwiklabs-gcp-01-3aff5ef1f764_cloudbuild/source/1606483297.219497-fc73065cc7c24588b5b7cbe4afb8f50c.tgz#1606483297852569...\n",
      "/ [1 files][  1.6 KiB/  1.6 KiB]                                                \n",
      "Operation completed over 1 objects/1.6 KiB.                                      \n",
      "BUILD\n",
      "Already have image (with digest): gcr.io/cloud-builders/docker\n",
      "Sending build context to Docker daemon  6.144kB\n",
      "Step 1/5 : FROM gcr.io/deeplearning-platform-release/base-cpu\n",
      "latest: Pulling from deeplearning-platform-release/base-cpu\n",
      "171857c49d0f: Pulling fs layer\n",
      "419640447d26: Pulling fs layer\n",
      "61e52f862619: Pulling fs layer\n",
      "20b22764011e: Pulling fs layer\n",
      "00244e2c5db1: Pulling fs layer\n",
      "07e452976526: Pulling fs layer\n",
      "9889ff203efe: Pulling fs layer\n",
      "05dad74dd489: Pulling fs layer\n",
      "abfc11aef694: Pulling fs layer\n",
      "00e45e47c0d1: Pulling fs layer\n",
      "5274e7716976: Pulling fs layer\n",
      "0dcd37ccffa2: Pulling fs layer\n",
      "8b7d4903c042: Pulling fs layer\n",
      "c65868d8f6c7: Pulling fs layer\n",
      "8497304a9c74: Pulling fs layer\n",
      "25b00734a98b: Pulling fs layer\n",
      "0e452e6fa8ed: Pulling fs layer\n",
      "20b22764011e: Waiting\n",
      "00244e2c5db1: Waiting\n",
      "07e452976526: Waiting\n",
      "9889ff203efe: Waiting\n",
      "05dad74dd489: Waiting\n",
      "abfc11aef694: Waiting\n",
      "00e45e47c0d1: Waiting\n",
      "5274e7716976: Waiting\n",
      "0dcd37ccffa2: Waiting\n",
      "8b7d4903c042: Waiting\n",
      "c65868d8f6c7: Waiting\n",
      "25b00734a98b: Waiting\n",
      "0e452e6fa8ed: Waiting\n",
      "8497304a9c74: Waiting\n",
      "419640447d26: Verifying Checksum\n",
      "419640447d26: Download complete\n",
      "61e52f862619: Verifying Checksum\n",
      "61e52f862619: Download complete\n",
      "171857c49d0f: Verifying Checksum\n",
      "171857c49d0f: Download complete\n",
      "07e452976526: Verifying Checksum\n",
      "07e452976526: Download complete\n",
      "00244e2c5db1: Verifying Checksum\n",
      "00244e2c5db1: Download complete\n",
      "05dad74dd489: Verifying Checksum\n",
      "05dad74dd489: Download complete\n",
      "9889ff203efe: Verifying Checksum\n",
      "9889ff203efe: Download complete\n",
      "abfc11aef694: Verifying Checksum\n",
      "abfc11aef694: Download complete\n",
      "5274e7716976: Verifying Checksum\n",
      "5274e7716976: Download complete\n",
      "00e45e47c0d1: Verifying Checksum\n",
      "00e45e47c0d1: Download complete\n",
      "0dcd37ccffa2: Verifying Checksum\n",
      "0dcd37ccffa2: Download complete\n",
      "8b7d4903c042: Verifying Checksum\n",
      "8b7d4903c042: Download complete\n",
      "c65868d8f6c7: Verifying Checksum\n",
      "c65868d8f6c7: Download complete\n",
      "8497304a9c74: Verifying Checksum\n",
      "8497304a9c74: Download complete\n",
      "0e452e6fa8ed: Verifying Checksum\n",
      "0e452e6fa8ed: Download complete\n",
      "20b22764011e: Verifying Checksum\n",
      "20b22764011e: Download complete\n",
      "171857c49d0f: Pull complete\n",
      "419640447d26: Pull complete\n",
      "25b00734a98b: Verifying Checksum\n",
      "25b00734a98b: Download complete\n",
      "61e52f862619: Pull complete\n",
      "20b22764011e: Pull complete\n",
      "00244e2c5db1: Pull complete\n",
      "07e452976526: Pull complete\n",
      "9889ff203efe: Pull complete\n",
      "05dad74dd489: Pull complete\n",
      "abfc11aef694: Pull complete\n",
      "00e45e47c0d1: Pull complete\n",
      "5274e7716976: Pull complete\n",
      "0dcd37ccffa2: Pull complete\n",
      "8b7d4903c042: Pull complete\n",
      "c65868d8f6c7: Pull complete\n",
      "8497304a9c74: Pull complete\n",
      "25b00734a98b: Pull complete\n",
      "0e452e6fa8ed: Pull complete\n",
      "Digest: sha256:f6c7ab6b9004322178cbccf7becb15a27fd3e2240e7335f5a51f8ff1861fd733\n",
      "Status: Downloaded newer image for gcr.io/deeplearning-platform-release/base-cpu:latest\n",
      " ---> 2e14efcab90e\n",
      "Step 2/5 : RUN pip install -U fire cloudml-hypertune scikit-learn==0.20.4 pandas==0.24.2\n",
      " ---> Running in 124cb20538c9\n",
      "Collecting fire\n",
      "  Downloading fire-0.3.1.tar.gz (81 kB)\n",
      "Collecting cloudml-hypertune\n",
      "  Downloading cloudml-hypertune-0.1.0.dev6.tar.gz (3.2 kB)\n",
      "Collecting scikit-learn==0.20.4\n",
      "  Downloading scikit_learn-0.20.4-cp37-cp37m-manylinux1_x86_64.whl (5.4 MB)\n",
      "Collecting pandas==0.24.2\n",
      "  Downloading pandas-0.24.2-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)\n",
      "Requirement already satisfied, skipping upgrade: six in /opt/conda/lib/python3.7/site-packages (from fire) (1.15.0)\n",
      "Collecting termcolor\n",
      "  Downloading termcolor-1.1.0.tar.gz (3.9 kB)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from scikit-learn==0.20.4) (1.18.5)\n",
      "Requirement already satisfied, skipping upgrade: scipy>=0.13.3 in /opt/conda/lib/python3.7/site-packages (from scikit-learn==0.20.4) (1.5.3)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2011k in /opt/conda/lib/python3.7/site-packages (from pandas==0.24.2) (2020.1)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.5.0 in /opt/conda/lib/python3.7/site-packages (from pandas==0.24.2) (2.8.1)\n",
      "Building wheels for collected packages: fire, cloudml-hypertune, termcolor\n",
      "  Building wheel for fire (setup.py): started\n",
      "  Building wheel for fire (setup.py): finished with status 'done'\n",
      "  Created wheel for fire: filename=fire-0.3.1-py2.py3-none-any.whl size=111005 sha256=1fa0bf8a2098be08c413be82d04b2bbd901ec65c75ca47e1a50cc1004d688576\n",
      "  Stored in directory: /root/.cache/pip/wheels/95/38/e1/8b62337a8ecf5728bdc1017e828f253f7a9cf25db999861bec\n",
      "  Building wheel for cloudml-hypertune (setup.py): started\n",
      "  Building wheel for cloudml-hypertune (setup.py): finished with status 'done'\n",
      "  Created wheel for cloudml-hypertune: filename=cloudml_hypertune-0.1.0.dev6-py2.py3-none-any.whl size=3987 sha256=e23f1677a9f67025327a7263fc88e1a93274dffa6b567b21cf74399552e12a36\n",
      "  Stored in directory: /root/.cache/pip/wheels/a7/ff/87/e7bed0c2741fe219b3d6da67c2431d7f7fedb183032e00f81e\n",
      "  Building wheel for termcolor (setup.py): started\n",
      "  Building wheel for termcolor (setup.py): finished with status 'done'\n",
      "  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=be8ac210173288401d5b6ecb62d3da0b4b84b015c9c7226da23248f04e9ebb87\n",
      "  Stored in directory: /root/.cache/pip/wheels/3f/e3/ec/8a8336ff196023622fbcb36de0c5a5c218cbb24111d1d4c7f2\n",
      "Successfully built fire cloudml-hypertune termcolor\n",
      "Installing collected packages: termcolor, fire, cloudml-hypertune, scikit-learn, pandas\n",
      "  Attempting uninstall: scikit-learn\n",
      "    Found existing installation: scikit-learn 0.23.2\n",
      "    Uninstalling scikit-learn-0.23.2:\n",
      "      Successfully uninstalled scikit-learn-0.23.2\n",
      "  Attempting uninstall: pandas\n",
      "    Found existing installation: pandas 1.1.4\n",
      "    Uninstalling pandas-1.1.4:\n",
      "      Successfully uninstalled pandas-1.1.4\n",
      "\u001b[91mERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.\n",
      "\n",
      "We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.\n",
      "\n",
      "visions 0.6.4 requires pandas>=0.25.3, but you'll have pandas 0.24.2 which is incompatible.\n",
      "pandas-profiling 2.8.0 requires pandas!=1.0.0,!=1.0.1,!=1.0.2,>=0.25.3, but you'll have pandas 0.24.2 which is incompatible.\n",
      "pandas-profiling 2.8.0 requires visions[type_image_path]==0.4.4, but you'll have visions 0.6.4 which is incompatible.\n",
      "\u001b[0mSuccessfully installed cloudml-hypertune-0.1.0.dev6 fire-0.3.1 pandas-0.24.2 scikit-learn-0.20.4 termcolor-1.1.0\n",
      "Removing intermediate container 124cb20538c9\n",
      " ---> 88ac1edc78ce\n",
      "Step 3/5 : WORKDIR /app\n",
      " ---> Running in b5abc852846e\n",
      "Removing intermediate container b5abc852846e\n",
      " ---> 07d1f11934df\n",
      "Step 4/5 : COPY train.py .\n",
      " ---> 8e4de5cbd8b0\n",
      "Step 5/5 : ENTRYPOINT [\"python\", \"train.py\"]\n",
      " ---> Running in b75b00bc25d6\n",
      "Removing intermediate container b75b00bc25d6\n",
      " ---> 50bb81723d4b\n",
      "Successfully built 50bb81723d4b\n",
      "Successfully tagged gcr.io/qwiklabs-gcp-01-3aff5ef1f764/trainer_image:latest\n",
      "PUSH\n",
      "Pushing gcr.io/qwiklabs-gcp-01-3aff5ef1f764/trainer_image:latest\n",
      "The push refers to repository [gcr.io/qwiklabs-gcp-01-3aff5ef1f764/trainer_image]\n",
      "4e6436a1e451: Preparing\n",
      "9b4f5636fb75: Preparing\n",
      "abf371a20d17: Preparing\n",
      "58c37b024800: Preparing\n",
      "093955a9f693: Preparing\n",
      "292c93aa8921: Preparing\n",
      "25e90c4f31bb: Preparing\n",
      "5ed5b5583a70: Preparing\n",
      "fed2ce1b9bf5: Preparing\n",
      "a2a7397c9263: Preparing\n",
      "135d5d53f509: Preparing\n",
      "28952c0fc305: Preparing\n",
      "1fff2aeddb5e: Preparing\n",
      "193419df8fce: Preparing\n",
      "9d1088ee89e7: Preparing\n",
      "98868f5e88f9: Preparing\n",
      "efa6a40d1ffb: Preparing\n",
      "7a694df0ad6c: Preparing\n",
      "3fd9df553184: Preparing\n",
      "805802706667: Preparing\n",
      "292c93aa8921: Waiting\n",
      "25e90c4f31bb: Waiting\n",
      "5ed5b5583a70: Waiting\n",
      "fed2ce1b9bf5: Waiting\n",
      "a2a7397c9263: Waiting\n",
      "135d5d53f509: Waiting\n",
      "28952c0fc305: Waiting\n",
      "1fff2aeddb5e: Waiting\n",
      "193419df8fce: Waiting\n",
      "9d1088ee89e7: Waiting\n",
      "98868f5e88f9: Waiting\n",
      "efa6a40d1ffb: Waiting\n",
      "7a694df0ad6c: Waiting\n",
      "3fd9df553184: Waiting\n",
      "805802706667: Waiting\n",
      "58c37b024800: Layer already exists\n",
      "093955a9f693: Layer already exists\n",
      "292c93aa8921: Layer already exists\n",
      "25e90c4f31bb: Layer already exists\n",
      "fed2ce1b9bf5: Layer already exists\n",
      "5ed5b5583a70: Layer already exists\n",
      "a2a7397c9263: Layer already exists\n",
      "135d5d53f509: Layer already exists\n",
      "28952c0fc305: Layer already exists\n",
      "1fff2aeddb5e: Layer already exists\n",
      "4e6436a1e451: Pushed\n",
      "9b4f5636fb75: Pushed\n",
      "9d1088ee89e7: Layer already exists\n",
      "193419df8fce: Layer already exists\n",
      "98868f5e88f9: Layer already exists\n",
      "efa6a40d1ffb: Layer already exists\n",
      "3fd9df553184: Layer already exists\n",
      "7a694df0ad6c: Layer already exists\n",
      "805802706667: Layer already exists\n",
      "abf371a20d17: Pushed\n",
      "latest: digest: sha256:6d738f69ab12c4bf78d7aee585222028d2ea55719c1f9a370e11704aeb5b99de size: 4499\n",
      "DONE\n",
      "--------------------------------------------------------------------------------\n",
      "\n",
      "ID                                    CREATE_TIME                DURATION  SOURCE                                                                                                      IMAGES                                                       STATUS\n",
      "5deceace-805d-406d-8e9e-bdfec88a4749  2020-11-27T13:21:38+00:00  3M27S     gs://qwiklabs-gcp-01-3aff5ef1f764_cloudbuild/source/1606483297.219497-fc73065cc7c24588b5b7cbe4afb8f50c.tgz  gcr.io/qwiklabs-gcp-01-3aff5ef1f764/trainer_image (+1 more)  SUCCESS\n"
     ]
    }
   ],
   "source": [
    "!gcloud builds submit --timeout 15m --tag $TRAINER_IMAGE trainer_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the base image for custom components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_NAME='base_image'\n",
    "TAG='latest'\n",
    "BASE_IMAGE='gcr.io/{}/{}:{}'.format(PROJECT_ID, IMAGE_NAME, TAG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating temporary tarball archive of 1 file(s) totalling 122 bytes before compression.\n",
      "Uploading tarball of [base_image] to [gs://qwiklabs-gcp-01-3aff5ef1f764_cloudbuild/source/1606483509.529902-02e717e4bbe24c6daf7e1902153a93a3.tgz]\n",
      "Created [https://cloudbuild.googleapis.com/v1/projects/qwiklabs-gcp-01-3aff5ef1f764/builds/bdc125ea-5d1a-4255-94b9-df5c5f8385bf].\n",
      "Logs are available at [https://console.cloud.google.com/cloud-build/builds/bdc125ea-5d1a-4255-94b9-df5c5f8385bf?project=1016448934670].\n",
      "----------------------------- REMOTE BUILD OUTPUT ------------------------------\n",
      "starting build \"bdc125ea-5d1a-4255-94b9-df5c5f8385bf\"\n",
      "\n",
      "FETCHSOURCE\n",
      "Fetching storage object: gs://qwiklabs-gcp-01-3aff5ef1f764_cloudbuild/source/1606483509.529902-02e717e4bbe24c6daf7e1902153a93a3.tgz#1606483510262206\n",
      "Copying gs://qwiklabs-gcp-01-3aff5ef1f764_cloudbuild/source/1606483509.529902-02e717e4bbe24c6daf7e1902153a93a3.tgz#1606483510262206...\n",
      "/ [1 files][  227.0 B/  227.0 B]                                                \n",
      "Operation completed over 1 objects/227.0 B.                                      \n",
      "BUILD\n",
      "Already have image (with digest): gcr.io/cloud-builders/docker\n",
      "Sending build context to Docker daemon  2.048kB\n",
      "Step 1/2 : FROM gcr.io/deeplearning-platform-release/base-cpu\n",
      "latest: Pulling from deeplearning-platform-release/base-cpu\n",
      "171857c49d0f: Pulling fs layer\n",
      "419640447d26: Pulling fs layer\n",
      "61e52f862619: Pulling fs layer\n",
      "20b22764011e: Pulling fs layer\n",
      "00244e2c5db1: Pulling fs layer\n",
      "07e452976526: Pulling fs layer\n",
      "9889ff203efe: Pulling fs layer\n",
      "05dad74dd489: Pulling fs layer\n",
      "abfc11aef694: Pulling fs layer\n",
      "00e45e47c0d1: Pulling fs layer\n",
      "5274e7716976: Pulling fs layer\n",
      "0dcd37ccffa2: Pulling fs layer\n",
      "8b7d4903c042: Pulling fs layer\n",
      "c65868d8f6c7: Pulling fs layer\n",
      "8497304a9c74: Pulling fs layer\n",
      "25b00734a98b: Pulling fs layer\n",
      "0e452e6fa8ed: Pulling fs layer\n",
      "20b22764011e: Waiting\n",
      "00244e2c5db1: Waiting\n",
      "07e452976526: Waiting\n",
      "9889ff203efe: Waiting\n",
      "05dad74dd489: Waiting\n",
      "abfc11aef694: Waiting\n",
      "00e45e47c0d1: Waiting\n",
      "5274e7716976: Waiting\n",
      "0dcd37ccffa2: Waiting\n",
      "8b7d4903c042: Waiting\n",
      "c65868d8f6c7: Waiting\n",
      "8497304a9c74: Waiting\n",
      "25b00734a98b: Waiting\n",
      "0e452e6fa8ed: Waiting\n",
      "61e52f862619: Verifying Checksum\n",
      "61e52f862619: Download complete\n",
      "419640447d26: Verifying Checksum\n",
      "419640447d26: Download complete\n",
      "171857c49d0f: Verifying Checksum\n",
      "171857c49d0f: Download complete\n",
      "07e452976526: Verifying Checksum\n",
      "07e452976526: Download complete\n",
      "00244e2c5db1: Verifying Checksum\n",
      "00244e2c5db1: Download complete\n",
      "05dad74dd489: Verifying Checksum\n",
      "05dad74dd489: Download complete\n",
      "9889ff203efe: Verifying Checksum\n",
      "9889ff203efe: Download complete\n",
      "abfc11aef694: Verifying Checksum\n",
      "abfc11aef694: Download complete\n",
      "5274e7716976: Verifying Checksum\n",
      "5274e7716976: Download complete\n",
      "00e45e47c0d1: Verifying Checksum\n",
      "00e45e47c0d1: Download complete\n",
      "8b7d4903c042: Verifying Checksum\n",
      "8b7d4903c042: Download complete\n",
      "0dcd37ccffa2: Verifying Checksum\n",
      "0dcd37ccffa2: Download complete\n",
      "c65868d8f6c7: Verifying Checksum\n",
      "c65868d8f6c7: Download complete\n",
      "8497304a9c74: Verifying Checksum\n",
      "8497304a9c74: Download complete\n",
      "20b22764011e: Verifying Checksum\n",
      "20b22764011e: Download complete\n",
      "0e452e6fa8ed: Verifying Checksum\n",
      "0e452e6fa8ed: Download complete\n",
      "171857c49d0f: Pull complete\n",
      "419640447d26: Pull complete\n",
      "61e52f862619: Pull complete\n",
      "25b00734a98b: Verifying Checksum\n",
      "25b00734a98b: Download complete\n",
      "20b22764011e: Pull complete\n",
      "00244e2c5db1: Pull complete\n",
      "07e452976526: Pull complete\n"
     ]
    }
   ],
   "source": [
    "!gcloud builds submit --timeout 15m --tag $BASE_IMAGE base_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile the pipeline\n",
    "\n",
    "You can compile the DSL using an API from the **KFP SDK** or using the **KFP** compiler.\n",
    "\n",
    "To compile the pipeline DSL using the **KFP** compiler."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set the pipeline's compile time settings\n",
    "\n",
    "The pipeline can run using a security context of the GKE default node pool's service account or the service account defined in the `user-gcp-sa` secret of the Kubernetes namespace hosting Kubeflow Pipelines. If you want to use the `user-gcp-sa` service account you change the value of `USE_KFP_SA` to `True`.\n",
    "\n",
    "Note that the default AI Platform Pipelines configuration does not define the `user-gcp-sa` secret."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: USE_KFP_SA=False\n",
      "env: BASE_IMAGE=gcr.io/qwiklabs-gcp-01-3aff5ef1f764/base_image:latest\n",
      "env: TRAINER_IMAGE=gcr.io/qwiklabs-gcp-01-3aff5ef1f764/trainer_image:latest\n",
      "env: COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/\n",
      "env: RUNTIME_VERSION=1.15\n",
      "env: PYTHON_VERSION=3.7\n"
     ]
    }
   ],
   "source": [
    "USE_KFP_SA = False\n",
    "\n",
    "COMPONENT_URL_SEARCH_PREFIX = 'https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/'\n",
    "RUNTIME_VERSION = '1.15'\n",
    "PYTHON_VERSION = '3.7'\n",
    "\n",
    "%env USE_KFP_SA={USE_KFP_SA}\n",
    "%env BASE_IMAGE={BASE_IMAGE}\n",
    "%env TRAINER_IMAGE={TRAINER_IMAGE}\n",
    "%env COMPONENT_URL_SEARCH_PREFIX={COMPONENT_URL_SEARCH_PREFIX}\n",
    "%env RUNTIME_VERSION={RUNTIME_VERSION}\n",
    "%env PYTHON_VERSION={PYTHON_VERSION}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the CLI compiler to compile the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "!dsl-compile --py pipeline/covertype_training_pipeline.py --output covertype_training_pipeline.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is the `covertype_training_pipeline.yaml` file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: argoproj.io/v1alpha1\n",
      "kind: Workflow\n",
      "metadata:\n",
      "  generateName: covertype-classifier-training-\n",
      "  annotations: {pipelines.kubeflow.org/kfp_sdk_version: 0.5.1, pipelines.kubeflow.org/pipeline_compilation_time: '2020-11-27T13:31:14.897418',\n",
      "    pipelines.kubeflow.org/pipeline_spec: '{\"description\": \"The pipeline training\n",
      "      and deploying the Covertype classifierpipeline_yaml\", \"inputs\": [{\"name\": \"project_id\"},\n",
      "      {\"name\": \"region\"}, {\"name\": \"source_table_name\"}, {\"name\": \"gcs_root\"}, {\"name\":\n",
      "      \"dataset_id\"}, {\"name\": \"evaluation_metric_name\"}, {\"name\": \"evaluation_metric_threshold\"},\n",
      "      {\"name\": \"model_id\"}, {\"name\": \"version_id\"}, {\"name\": \"replace_existing_version\"},\n"
     ]
    }
   ],
   "source": [
    "!head covertype_training_pipeline.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the pipeline package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(500)\n",
      "Reason: Internal Server Error\n",
      "HTTP response headers: HTTPHeaderDict({'Content-Length': '1461', 'Content-Type': 'text/html; charset=utf-8', 'Date': 'Fri, 27 Nov 2020 13:31:23 GMT', 'Vary': 'Origin', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'SAMEORIGIN', 'X-Powered-By': 'Express', 'X-Xss-Protection': '0', 'Set-Cookie': 'S=cloud_datalab_tunnel=QvLfMb8LotIqUres7gFDB8s6CCA-KlKivktVb5cjpv0; Path=/; Max-Age=3600'})\n",
      "HTTP response body: \n",
      "<!DOCTYPE html>\n",
      "<html lang=en>\n",
      "  <meta charset=utf-8>\n",
      "  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n",
      "  <title>Error 500 (Internal Server Error)!!1</title>\n",
      "  <style>\n",
      "    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n",
      "  </style>\n",
      "  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n",
      "  <p><b>500.</b> <ins>That’s an error.</ins>\n",
      "  <p>  <ins>That’s all we know.</ins>\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "PIPELINE_NAME='covertype_continuous_training'\n",
    "\n",
    "!kfp --endpoint $ENDPOINT pipeline upload \\\n",
    "-p $PIPELINE_NAME \\\n",
    "covertype_training_pipeline.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting pipeline runs\n",
    "\n",
    "You can trigger pipeline runs using an API from the KFP SDK or using KFP CLI. To submit the run using KFP CLI, execute the following commands. Notice how the pipeline's parameters are passed to the pipeline run.\n",
    "\n",
    "### List the pipelines in AI Platform Pipelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| Pipeline ID                          | Name                                            | Uploaded at               |\n",
      "+======================================+=================================================+===========================+\n",
      "| defa3c60-637b-4332-88b9-d8647c2aec84 | covertype_continuous_training                   | 2020-11-27T13:29:43+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| b9d5fe74-7c0a-4350-897c-27b373642fed | tfx_covertype-v2                                | 2020-11-24T15:46:37+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| 4e656e01-47dc-45c3-9bb6-df557dba99ba | covertype_continuous_training_test              | 2020-11-24T15:28:38+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| 892cd8cf-f8a2-4b5f-944e-ebb5d4e1d518 | tfx_covertype-test                              | 2020-11-23T13:31:02+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| 49d8f17f-b019-4e87-95a3-0d6a32924c83 | [Tutorial] DSL - Control structures             | 2020-11-23T11:20:41+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| 7511670b-9a7c-42db-b880-44a193d17d3a | [Tutorial] Data passing in python components    | 2020-11-23T11:20:40+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| 24507f88-1f33-410c-8f4f-4be1c8952650 | [Demo] TFX - Iris classification pipeline       | 2020-11-23T11:20:38+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| 3ac9118d-4b07-4559-a5fb-57129b176263 | [Demo] TFX - Taxi tip prediction model trainer  | 2020-11-23T11:20:37+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n",
      "| c98150b3-f79b-4823-b267-8baab62f7c59 | [Demo] XGBoost - Training with confusion matrix | 2020-11-23T11:20:36+00:00 |\n",
      "+--------------------------------------+-------------------------------------------------+---------------------------+\n"
     ]
    }
   ],
   "source": [
    "!kfp --endpoint $ENDPOINT pipeline list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit a run\n",
    "\n",
    "Find the ID of the `covertype_continuous_training` pipeline you uploaded in the previous step and update the value of `PIPELINE_ID` .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "PIPELINE_ID='defa3c60-637b-4332-88b9-d8647c2aec84'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "EXPERIMENT_NAME = 'Covertype_Classifier_Training'\n",
    "RUN_ID = 'Run_001'\n",
    "SOURCE_TABLE = 'covertype_dataset.covertype'\n",
    "DATASET_ID = 'splits'\n",
    "EVALUATION_METRIC = 'accuracy'\n",
    "EVALUATION_METRIC_THRESHOLD = '0.69'\n",
    "MODEL_ID = 'covertype_classifier'\n",
    "VERSION_ID = 'v01'\n",
    "REPLACE_EXISTING_VERSION = 'True'\n",
    "\n",
    "GCS_STAGING_PATH = '{}/staging'.format(ARTIFACT_STORE_URI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run 4fdb59af-17b7-4136-83be-e7eeec23248b is submitted\n",
      "+--------------------------------------+---------+----------+---------------------------+\n",
      "| run id                               | name    | status   | created at                |\n",
      "+======================================+=========+==========+===========================+\n",
      "| 4fdb59af-17b7-4136-83be-e7eeec23248b | Run_001 |          | 2020-11-27T13:32:24+00:00 |\n",
      "+--------------------------------------+---------+----------+---------------------------+\n"
     ]
    }
   ],
   "source": [
    "!kfp --endpoint $ENDPOINT run submit \\\n",
    "-e $EXPERIMENT_NAME \\\n",
    "-r $RUN_ID \\\n",
    "-p $PIPELINE_ID \\\n",
    "project_id=$PROJECT_ID \\\n",
    "gcs_root=$GCS_STAGING_PATH \\\n",
    "region=$REGION \\\n",
    "source_table_name=$SOURCE_TABLE \\\n",
    "dataset_id=$DATASET_ID \\\n",
    "evaluation_metric_name=$EVALUATION_METRIC \\\n",
    "evaluation_metric_threshold=$EVALUATION_METRIC_THRESHOLD \\\n",
    "model_id=$MODEL_ID \\\n",
    "version_id=$VERSION_ID \\\n",
    "replace_existing_version=$REPLACE_EXISTING_VERSION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where\n",
    "\n",
    "- EXPERIMENT_NAME is set to the experiment used to run the pipeline. You can choose any name you want. If the experiment does not exist it will be created by the command\n",
    "- RUN_ID is the name of the run. You can use an arbitrary name\n",
    "- PIPELINE_ID is the id of your pipeline. Use the value retrieved by the   `kfp pipeline list` command\n",
    "- GCS_STAGING_PATH is the URI to the GCS location used by the pipeline to store intermediate files. By default, it is set to the `staging` folder in your artifact store.\n",
    "- REGION is a compute region for AI Platform Training and Prediction. \n",
    "\n",
    "You should be already familiar with these and other parameters passed to the command. If not go back and review the pipeline code.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitoring the run\n",
    "\n",
    "You can monitor the run using KFP UI. Follow the instructor who will walk you through the KFP UI and monitoring techniques.\n",
    "\n",
    "To access the KFP UI in your environment use the following URI:\n",
    "\n",
    "https://[ENDPOINT]\n",
    "\n",
    "\n",
    "**NOTE that your pipeline run may fail due to the bug in a BigQuery component that does not handle certain race conditions. If you observe the pipeline failure, retry the run from the KFP UI**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=-1>Licensed under the Apache License, Version 2.0 (the \\\"License\\\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.</font>"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m59",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m59"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
